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PRIORITIZING  CANDIDATE  DECISION  AIDS  FOR  TACTICAL  APPLICATIONS: 

REPORT  OF  A  WORKSHOP 


INTRODUCTION 


■  This  report  describes  a  methodology  for  prioritizing  candidate  decision 
aids  which  can  be  developed  using  the  techniques  of  artificial  itelligence 
(AI).  The  methodology  was  tested  in  an  action  officer  Prioritization  Workshop 
conducted  at  Ft.  Leavenworth  in  November  1986.  This  report  details  the  method¬ 
ology,  results  of  the  prioritization,  limitations  of  and  lessons  learned  from 
the  Workshop,  prioritizing  issues,  and  general  recommendations  for  a 
prioritizing  methodology.  _ ^ 


Artificial  intelligence  has  been  identified  by  the  Department  of  Defense 
as  one  of  the  most  critical  technologies  to  pursue  for  the  remainder  of  this 
century  (D'Ambrosio,  1985).  Due  to  its  high  priority,  some  efficient  and  sys¬ 
tematic  method  is  needed  to  identify  potential  AI  applications  and  projects  and 
choose  among  them  for  funding.  If  the  bases  for  development  decisions  are  not 
explicitly  chosen,  the  funding  decisions  will  still  be  made,  but  they  will  be 
based  on  implicit  criteria  such  as  vendors'  recommendations,  political  pres¬ 
sures,  or  a  new  technology  that  catches  the  developer's  interest.  Not  only  is 
it  important  in  this  time  of  austerity  to  get  the  most  for  the  money  spent, 
wasting  time  on  projects  that  do  not  contribute  to  overall  Army  goals  can  put 
us  behind  in  the  technology  race  with  potential  adversaries.  Further,  more 
effort  spent  choosing  between  candidate  aids  will  result  in  fewer  abandoned 
developments.  This  report  describes  and  assesses  one  such  prioritizing 
methodology. 


Artificial  intelligence  is  a  group  of  computer  based  technologies  which 
produce  results  commonly  thought  to  require  human  intelligence.  AI  is  a 
loosely  defined  field  and  includes  a  number  of  technologies:  expert  systems, 
natural  language  understanding,  robotics,  artificial  vision,  sound  sensing  and 
understanding,  learning,  and  information  fusion.  Because  expert  systems  are 
particularly  relevant  to  military  tactical  applications,  this  paper  will  be 
limited  to  expert  systems.  Expert  systems  are  computer  programs  that  are  based 
on  the  knowledge  that  underlies  human  expertise.  The  expert  has  two  kinds  of 
knowledge:  facts  and  theories  found  in  textbooks  and  private  knowledge  gained 

through  many  years  of  experience.  This  private  knowledge  is  not  formalized  but 
consists  of  rules  of  thumb,  short  cuts,  and  educated  guesses  that  the  expert 
has  found  to  work  in  the  past.  These  educated  guesses,  or  heuristics,  are 
especially  appropriate  for  dealing  with  problem  areas  characterized  by  unrelia¬ 
ble  and  incomplete  data.  Exact  problem  solving  algorithms  usually  make  certain 
assumptions  about  the  reliability,  completeness  and  the  amount  of  error  in  the 
data  they  use.  However,  many  real  world  problems  originate  in  complex,  chang¬ 
ing  situations  with  information  that  is  uncertain  and  incomplete.  Battlefield 
situations  are  just  such  problems,  and  expert  systems  hold  great  promise  for 
decision  making  support  in  the  uncertain,  high  stress,  information  overloaded 
environment  of  tactical  battle  management. 
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BACKGROUND 


In  December  1985,  the  Fort  Leavenworth  TRADOC^  Analysis  Center  (TRAC,  for¬ 
merly  CAORA)  released  the  report  "G3  Analysis"  (USA  CAORA,  1985),  which 
prioritized  potential  tactical  decision  aids  that  would  use  optimization,  simu¬ 
lation,  or  decision  analysis  techniques.  A1  techniques  were  identified  as 
having  potential  for  use  in  the  development  of  decision  aids,  but  the  report 
did  not  evaluate  the  application  of  A1  to  the  aids.  The  Combined  Arms  Combat 
Developments  Activity  -  Command,  Control,  Communications  and  Intelligence 
(CACDA-C3I)  Directorate  requested  that  the  Arny  Research  Institute  (ARI)  Fort 
Leavenworth  Field  Unit:  (1)  evaluate  the  potential  for  the  application  of  AI 
to  the  development  of  command  and  control  decision  aids,  (2)  develop  a 
prioritizing  methodology  for  AI  based  aids,  (3)  prioritize  a  set  of  tactical 
decision  aids,  and  (4)  recommend  AI  tools  that  could  be  used  to  build  the  aids. 


The  first,  second,  and  third  of  these  tasks  were  addressed  in  a  CACDA-C3I 
Action  Officer  Prioritization  Workshop,  supported  by  ARI,  in  November  1986. 

This  report  documents  the  methodology  and  results  of  the  CACDA  Decision  Aids 
Prioritization  Workshop.  The  Workshop  consisted  of  two  phases  and  prioritized 
a  subset  of  the  aiding  opportunities  identified  in  the  1985  TRAC  "G3  Analysis". 

The  TRAC  "G3  Analysis"  report  analyzed  G3  functions,  identified  53 
opportunities  to  aid  G3  performance  through  the  use  of  computer  applications, 
and  prioritized  these  53  opportunities.  Appendix  A  describes  the  53  aids  and 
Appendix  B  shows  the  results  of  the  "G3  Analysis"  report  prioritizations.  The 
first  part  of  the  Prioritization  Workshop  selected  a  subset  of  23  aiding  oppor¬ 
tunities  from  the  53  "G3  Analysis"  report  aids.  These  23  aids  were  those  for 
which  AI  could  be  appropriately  used  in  the  design  of  the  aid.  The  second  part 
of  the  Prioritization  Workshop  was  a  multi-attribute  analysis  in  which  the  23 
aids  selected  in  the  first  part  were  rated  on  the  attributes  of  importance  and 
feasibility. 

The  remainder  of  this  report  presents  the  methodology,  results,  and  les¬ 
sons  learned  in  the  CACDA  AI  Decision  Aid  Prioritization  Workshop,  and  makes 
recommendations  for  future  prioritization  efforts. 

The  fourth  task,  to  recommend  AI  tools  for  aid  development,  is  treated  in 
the  report,  A  Survey  of  Expert  System  Development  Tools,  (Liebhaber  &  Riedel, 
1987).  In  this  report,  each  of  93  tools  is  described  in  terms  of  its  knowl¬ 
edge  representation,  control  and  inference,  certainty  management,  hypothesis 
handling,  and  knowledge  acquisition  features.  The  cost,  availability,  vendor 
support,  user  interface  devices  and  hardware  and  software  requirements  of  each 
tool  are  also  listed. 
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Training  and  Doctrine  Command. 
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A  forthcoming  ARI  report  (Riedel, in  preparation),  which  is  also  related  to 
this  project,  will  discuss  prioritizing  methodologies  in  general  and  make 
recommendations  for  developing  an  improved  methodology.  The  report  will 
discuss  in  detail  criteria  for  choosing  expert  system  applications,  and  alter¬ 
native  prioritization  methodologies. 
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PROCEDURE 


Purpose .  The  Workshop  had  two  main  purposes:  (1)  to  identify  and  make  a 
preliminary  prioritization  of  tactical  decision  aids  suitable  for  development 
using  AI  technology,  and  (2)  to  test  a  methodology  for  prioritizing  A1  based 
decision  aids. 

Participants «  Action  officers  were  invited  from  organizations  currently  doing 
tactical  decision  aid  work.  Attendees  included  personnel  from  TRAC,  Command  and 
General  Staff  College,  Center  for  Army  Tactics,  Combined  Arms  Combat  Develop¬ 
ments  Activity,  Army  Research  Institute,  Mitre  Corporation,  Army  Signal  Cen¬ 
ter,  Army  Intelligence  Center,  and  the  Soldier  Support  Center. 

Agenda.  The  Workshop  covered  one  and  a  half  days.  The  first  half  day  was 
used  to  brief  the  participants  on  background  information  they  would  need  to 
make  the  prioritization  judgments. 

Briefings  included: 

1.  The  Army  Tactical  Command  and  Control  System. 

2.  Introduction  to  Artificial  Intelligence. 

3.  The  Airland  Battle  Management  Program  and  Tactical  Decision  Aids 
currently  under  development. 

4.  The  Joint  Army  Materiel  Command/TRADOC  AI  Working  Groups. 

5.  The  Combined  Arms  Center  (CAC)/ARI  Experimental  Development  Demonstra¬ 
tion  and  Integration  Center.  (See  Packard,  McKeown,  Falleson,  Solick  &  Halpin, 
1987,  for  a  detailed  description.) 

6.  The  "G3  Decision  Aid  Analysis" 

7.  Prioritizing  AI  Tactical  Decisions 

After  the  briefings,  participants  then  broke  into  two  Working  Groups.  The 
first  addressed  the  prioritization  of  AI  tactical  decision  aids  and  the  second 
addressed  the  steps  in  the  development  of  AI  software  from  idea  to  implementa¬ 
tion.  These  meetings  extended  over  one  day.  This  report  describes  the  activi¬ 
ties  of  the  first  Working  Group. 


Prioritization  Methodology 

ARI  developed  a  methodology  to  extend  the  procedure  used  in  the  "G3 
Analysis"  report.  To  do  this  it  added  an  expert  system  filter  prior  to  the 
multi-attribute  utility  analysis  (MAUA)  prioritization  methodology  used  by  the 
"G3  Analysis"  report.  Figure  1  shows  the  revised  methodology.  The  first  step 
is  to  identify  a  set  of  candidate  aids.  The  second  step  is  to  determine 
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Figure  1 .  Prioritization  methodology  for  Al  tactical  decision  aids. 


whether  the  application  of  expert  system  technology  is  appropriate  for  develop¬ 
ment  of  each  aid.  Expert  systems  are  clearly  Inappropriate  for  some  aids,  and 
marginally  appropriate  for  others.  The  third  step  is  to  apply  a  multi-attribute 
utility  prioritization  methodology  to  those  aids  for  which  AI  is  appropriate. 

STEP  1.  Identify  Candidate  Aids. 

This  is  a  requirements  analysis,  where  the  needs  of  the  intended  users  are 
analyzed  and  aids  that  could  meet  those  needs  are  identified.  The  "G3  Analy¬ 
sis"  presents  the  results  of  a  requirements  analysis  where  53  candidate  aids 
were  identified.  The  Workshop  bypassed  Step  1  and  used  the  53  aids  identified 
in  the  "G3  Analysis"  as  input  to  Step  2. 

STEP  2:  Determine  the  Appropriateness  of  Expert  System  Technology  for  Each  Aid. 

Not  all  the  problem  areas  that  could  benefit  from  automated  systems  are 
appropriate  areas  for  the  application  of  expert  systems.  A  review  of  the 
literature  in  expert  systems  was  conducted  to  derive  criteria  to  use  in 
selecting  applications  for  expert  systems.  From  this  review  the  following 
emerged  as  necessary  characteristics  of  the  problem  area  and  experts  in  that 
area: 

o  Experts  exist  in  the  problem  area. 

o  Experts  agree  on  the  correct  solution  and  how  to  proceed. 

o  Experts  can  describe  their  procedures. 

o  Other  aiding  methods  would  not  be  satisfactory. 

Three  Army  officers,  two  lieutenant  colonels  and  one  major  who  were  famil¬ 
iar  with  G3  tactical  decisions  and  ES  technology,  were  asked  to  evaluate  each 
of  the  53  candidate  aids  on  the  above  criteria. 

Twenty-two  of  the  original  53  aids  met  all  the  criteria  and  were  passed  on 
to  the  next  step.  The  raters  also  added  one  aid.  Situation  Assessment,  to  the 
list  of  potential  AI  aids.  Table  1  lists  the  23  aids  that  were  identified  in 
this  step. 

STEP  3;  Compare  Candidate  Aids. 

Multi-attribute  Utility  Analysis  (MAUA)  was  used  to  compare  the  aids  with 
each  other.  MAUA  is  a  technique  for  aiding  decision  problems  that  are  charac¬ 
terized  by  multiple  objectives.  It  is  based  on  the  principle  of  decomposition. 
The  objectives  in  choosing  an  aid  for  development  are  broken  into  component 
parts,  analyzed,  and  then  combined  to  yield  composite  judgments  for  each  aid 
which  can  then  be  used  to  rank  order  the  aids.  There  are  a  number  of  versions 
of  MAUA  (cf.  Edwards,  1977;  Keeney  &  Raiffa,  1976;  Pitz  &  McKillip,  1984). 

This  version  is  similar  to  that  of  Edwards  (1977). 
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Table  1 


Tactical  Decision  Aiding  Opportunities  Suitable  for 
Expert  System  Development 


Forecast  Tube  Replacement 
Controlled  Supply  Rate  (CSR) 

Prescribed  Chemical  Load  (PCL) 

Force  Movement  Analyzer 
Fuel  Consumption  Rates 
Air  Movement  Planner 
Denial  Preparation 

Compare  Alternative  Courses  of  Action 
Obstacle  Preparation 
Chemical  Effects  Preparation 
Expenditure  Rates  (Fire  Support  Annex) 

Basic  Load  Allocations 
Nuclear  Effects  Prediction 
Aircraft  Asset  Analyzer 

Priorities/Allocation  (Air  Defense  Annex) 

Rear  Area  Protection  Capabilities 
Fallout  Prediction  (Nuclear) 

NBC  Effects  Evaluation 
Assign  Psychological  Operations  Assets 
Air  Movement  Analyzer 
Situation  Assessment 

Optimal  Atomic  Demolition  Munitions  Employment 
Nuclear  Effects  Prediction 


MAUA  is  especially  appropriate  for  competing  objectives.  For  example,  one 
objective  in  choosing  an  aid  for  development  may  be  to  develop  an  aid  for  an 
important  problem,  that  is,  where  there  will  be  a  great  increase  in  the  effec¬ 
tiveness  of  the  decision  when  using  the  aid.  One  may  also  wish  to  maximize 
the  probability  of  successfully  developing  the  aid.  If  there  is  a  lot  of  room 
for  improvement  using  the  aid  this  means  the  problem  is  probably  complex  and 
difficult.  This  also  means  that  well  established  procedures  for  solving  the 
problem  have  not  been  developed,  and  it  is  likely  that  the  AI  technology  for 
developing  the  aid  may  still  be  rudimentary.  The  j  ;rson  prioritizing  the  aids 
must  decide  how  much  importance  he  is  willing  to  trade  off  for  a  decreased  risk 
in  implementing  the  aid.  MAUA  provides  procedures  for  making  such  tradeoffs. 

MAUA  Prioritization  Steps 

1.  Identify  the  criteria  for  ranking  the  aids.  These  are  the  aspects  of 
the  aids  the  decision  maker  wants  to  maximize  with  his/her  rankings.  In  this 
case,  criteria  were  taken  from  "G3  Analysis",  which  specified 
Importance  of  the  aid  and  feasibility  of  implementation  as  aspects  of  the  aids 
that  would  determine  their  rankings. 
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2.  Identify  the  subcriteria  and  how  they  are  to  be  measured.  Subcriteria 
are  component  parts  of  each  criteria.  The  "G3  Analysis"  report  defined  impor¬ 
tance  as  being  composed  of:  (a)  time  and  quality  savings  in  using  the  aid, 

(b)  frequency  with  which  the  aid  will  be  used,  and  (c)  number  of  Commander's 
Critical  Information  Requirements  (CCIR)  the  aid  incorporates.  Feasibility  was 
broken  down  into:  (a)  economical  -  estimated  cost  of  developing  and  implement¬ 
ing  the  aid,  (b)  technical  -  probability  that  the  needed  technology  exists  to 
develop  the  aid  within  the  next  five  years,  and  (c)  operational  -  probability 
that  the  aid  will  be  successfully  fielded. 

3.  Weight  the  criteria  and  subcriteria.  The  weight  reflects  how  much 
each  criteria  and  subcriteria  contributes  to  the  overall  desirability  of  the 
aid.  The  weights  used  in  the  Workshop  were  taken  from  the  "G3  Analysis", 
which  used  the  Saaty  method  of  weighting  (Saaty,  1980).  In  this  method  raters 
are  asked  to  make  comparisons  of  relative  importance  between  pairs  of  criteria 
and  similarly  between  pairs  of  subcriteria.  The  weights  selected  are  shown  in 
Figure  2. 

4.  Obtain  a  value  for  each  candidate  aid  on  each  subcriteria.  This  step 
was  the  primary  activity  of  the  Workshop.  Twelve  raters  participated  in  the 
Prioritization  Workshop.  Raters  were  given  a  rating  sheet  for  each  of  the  23 
candidate  aids  and  asked  to  rate  each  aid  on  a  scale  of  one  to  ten  on  each 
subcriterion.  Appendix  C  shows  a  sample  rating  sheet.  Average  rating  across 
participants  were  obtained  for  each  aid  on  each  attribute.  A  measure  of  rater 
agreement,  Chronbach's  alpha  (SPSSX  Users'  Manual,  1983),  was  calculated  for 
each  aid.  Appendix  D  shows  the  interrater  correlations.  Examination  of  this 
table  shows  that  for  many  of  the  aids  there  was  little  or  no  agreement  among 
the  raters  on  the  ratings  that  were  assigned  to  the  subcriteria.  In  cases 
where  there  was  a  wide  variation  between  raters'  scores,  raters  discussed  the 
reasons  for  their  ratings  and  arrived  at  a  consensus.  For  those  ratings  that 
were  discussed,  the  consensus  score  was  used  in  Step  5  for  X(ij)  rather  than  an 
averaged  rating. 

5.  Aggregate  the  weighted  scores  for  each  option.  A  total  score,  for 
each  candidate  aid  was  calculated  using  the  following  formula: 


T(.)  =  ■JwCj)  x(ij) 


Where  T(i)  is  the  total  score  for  candidate  aid  i,  w(j)  is  th'-  weight  of  sub- 
criterion  j,  x(ij)  is  the  averaged  rating  of  aid  i  on  subcriterion  j. 

The  average  rating  of  each  aid  on  each  attribute  was  weighted  and  summed 
to  yield  a  total  score  for  each  aid. 
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LEVEL  1 : 
CRITERIA 


LEVEL  2: 
SUB¬ 
CRITERIA 


Figure  2. 


PRIORITIZED  SCORE 
TOTAL  VALUE  OF  AID 


QUALITY 

SAVINGS 

WT=.36 


Hierarchy  of  aid  criteria. 
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6.  Conduct  a  sensitivity  analysis.  A  sensitivity  analysis  makes  changes 
in  the  weights  and  possibly  criteria  and  subcriteria  to  see  how  these  changes 
affect  the  rankings.  This  analysis  enables  an  evaluation  of  the  robustness  of 
the  rankings  and  an  examination  of  which  variables  are  critical  in  producing 
the  rankings  which  were  obtained.  In  the  Workshop,  the  sensitivity  analysis 
was  done  by  changing  the  weights  of  the  criteria  and  subcriteria.  Weighted  and 
unweighted  total  scores  were  calculated,  and  the  results  were  presented  to  the 
participants.  Where  there  were  large  discrepancies  between  the  weighted  and 
unweighted  scores  for  an  aid,  possible  reasons  for  the  discrepancies  were  dis¬ 
cussed. 

7.  Rank  the  aids.  Candidate  aids  are  ranked  based  on  their  aggregated 
weighted  scores  T(i). 
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RESULTS 


Table  2  shows  the  twelve  top  ranking  aids  and  their  ranks.  The  top  four 
aids  are  Alternative  Courses  of  Action  Analyzer,  Force  Movement  Analyzer,  Ob¬ 
stacle  Preparation,  and  Fuel  Consumption  Rate  Analyzer.  Descriptions  of  these 
aids  can  be  found  in  Appendix  A. 


Table  2 

Prioritization  Rank  of  Top  Twelve  Candidate  Aids 


Aid  Descriptors 

Rank 

Absolute 

Weighted 

Compare  Alternative  Courses  of 

1 

1 

Action  (COA) 

Force  Movement  Analyzer 

2 

2 

Obstacle  Preparation 

3 

4 

Fuel  Consumption  Rates 

4 

3 

Denial  Preparation 

5 

5 

Basic  Load 

6 

12 

Priorities  Allocation 

7 

6 

Chemical  Effects  Prediction 

8 

10 

Optimal  Atomic  Demolition 

9 

9 

Munitions  Employment 

Rear  Area  Protection  Capabilities 

10 

7 

Situation  Assessment 

11 

11 

Air  Movement  Analyzer 

12 

8 

A  sensitivity  analysis,  comparing  weighted  and  unweighted  rankings,  shows 
that  the  set  of  the  twelve  highest  ranking  aids  is  the  same  for  both  the 
weighted  and  unweighted  rankings.  Similarly,  the  sets  of  five  highest  ranking 
aids  is  the  same.  This  suggests  that  these  sets  of  the  top  five  and  the  top 
twelve  aids  are  relatively  robust. 

There  are,  however,  several  cases  with  large  discrepancies  between  the 
weighted  and  unweighted  ranks  of  an  aid.  Aids  with  large  discrepancies  are  Air 
Movement  Analyzer,  Basic  Load,  and  Rear  Area  Protection  Capabilities.  The 
reason  for  these  discrepancies  is  apparent  when  the  raw  data  are  examined.  The 
CCIR  subcriterion  has  a  relatively  high  weight;  it  accounts  for  over  a  third  of 
the  total  weight.  Both  Air  Movement  Analyzer  and  Rear  Area  Protection  Capa¬ 
bilities  have  high  ratings  on  CCIR  and  low  ratings  on  the  other  attributes.  The 
high  CCIR  score,  with  its  high  weighting,  raises  the  ranking  of  these  aids 
compared  to  the  unweighted  ranking.  Basic  Load  on  the  other  hand  has  a  very  low 
CCIR  score  and  high  scores  on  the  other  variables.  Weighting  the  CCIR  score 
makes  its  low  value  have  more  of  an  effect  and  drags  down  the  relative  total 
score  of  the  Basic  Load  aid.  These  discrepancies  suggest  that  both  the  use  and 
weight  of  the  subcriterion  "CCIR"  should  be  re-examined. 
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The  interrater  reliability  of  the  Workshop  participants'  ratings  vary  from 
.875  to  0.  See  Appendix  D.  However,  the  four  top  rated  aids  all  had  relia¬ 
bilities  over  .62,  an  acceptable  reliability  for  ratings  of  this  type.  Fur¬ 
ther,  the  top  rated  aid.  Compare  Alternate  Courses  of  Action,  had  an  interrater 
reliability  of  .875.  This  means  that  V/orkshop  participants  tended  to  agree  on 
the  top  four  aids,  and  to  highly  agree  that  the  top  aid  should  be  Compare 
Alternate  Courses  of  Action. 
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PROBLEMS/LIMITATIONS  OF  THE  WORKSHOP 


Raters .  It  is  very  important  to  use  raters  with  expertise  in  the  areas 
being  judged.  In  the  Workshop,  raters  with  little  background  in  AI  were  asked 
to  make  technical  feasibility  and  cost  ratings.  Similarly,  personnel  with 
little  experience  in  G3  operations  were  asked  to  rate  the  aids  on  importance. 

In  the  Workshop,  raters  were  asked  to  omit  the  ratings  on  a  particular  dimen¬ 
sion  if  they  didn't  feel  qualified  to  do  the  ratings.  However,  as  the  many 
zero  interrater  correlations  in  Appendix  D  indicate,  many  raters  did  not  have 
the  required  expertise  with  which  to  make  the  ratings.  There  is  no 
reason  why  different  raters  could  not  provide  data  for  the  different  ratings 
that  are  required. 

In  this  prioritization  methodology,  four  types  of  ratings  are  required:  (1) 
ratings  of  the  appropriateness  of  ES  technology  for  the  candidate  aids'  devel¬ 
opment  in  Step  2;  (2)  the  assignment  of  weights  to  the  criteria  and  subcriteria 
in  Step  5  of  the  MAUA  prioritization;  (3)  ratings  of  the  potential  contribution 
of  the  aid  to  improvement  in  battlefield  outcome;  and  (4)  ratings  of  the  feasi¬ 
bility  of  developing  and  implementing  the  aid.  Each  of  these  types  of  ratings 
requires  a  different  area  of  expertise.  The  first  type  requires  expertise  in 
both  AI  technology  and  military  tactical  operations.  The  rater  should  be 
knowledgeable  about  tactical  operations  so  that  he  understands  what  functions 
the  aid  would  perform.  He  should  have  a  background  in  AI  so  that  he  can  judge 
whether  AI  technologies  can  best  support  those  functions.  The  three  officers 
rating  the  appropriateness  of  ES  technology  had  both  of  these  types  of  exper¬ 
tise.  The  second  type  of  rating,  criterion  and  subcriterion  weights,  is  best 
made  by  those  who  will  make  the  decision  of  which  aids  to  develop  and  imple¬ 
ment.  These  are  not  judgements  of  fact  or  information  but  judgements  of  value 
and  preference.  That  is,  the  rater  must  decide  what  he  wants  to  accomplish 
with  his  choice  of  aids  to  develop.  The  third  type  of  rating,  rating  of  the 
importance  of  the  candidate  aid,  requires  expertise  in  tactical  operations  and 
the  fourth  type,  ratings  of  feasibility  requires  expertise  in  AI.  If  the  same 
raters  do  not  have  the  expertise,  and  in  the  case  of  the  weighting  judgements, 
the  role  to  make  all  of  these  ratings,  then  different  raters  should  be  used  to 
make  the  different  judgements  required  by  the  methodology. 

In  the  Workshop,  different  raters  made  the  ratings  of  ES  appropriateness 
and  the  feasibility  and  importance  criteria  ratings.  As  the  low  interrater 
agreement  correlations  indicate,  all  the  raters  did  not  have  the  needed  exper¬ 
tise  to  do  the  subcriteria  ratings.  However,  discussing  the  ratings  for  which 
there  was  a  large  amount  of  disagreement  helped  mitigate  the  effects  of  using 
the  wrong  raters. 

Rating  Anchors.  The  rating  scales  need  anchors.  For  example,  the  meaning 
of  cost  is  unclear.  What  is  high  to  one  person  may  be  intermediate  to  another. 
If  high  cost  means  $500,000  to  one  person  and  3  million  to  another,  an  aid 
estimated  to  cost  $500,000  might  be  rated  as  a  10  by  the  first  and  a  one  by  the 
other. 

CCIR.  This  subcriterion  measures  the  number  of  major  subcategories  of  CCIR 
which  the  aid  supports.  It  was  included  in  the  TRAC  "G3  Analysis"  because  the 
authors  felt  that  aid  utility  increases  with  increasing  production/support  of 
CCIR.  Workshop  participants  thought  that  some  CCIR  may  be  more  critical  than 
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others  and  contribute  more  to  the  total  score.  There  should  not  be,  therefore, 
a  direct  relationship  between  number  of  CCIR  and  the  aid's  prioritization 
score.  Participants  thought  the  method  of  calculating  this  CCIR  score  should 
be  re-examined.  Further,  many  participants  thought  the  large  weight  of  this 
subcriterion  was  inappropriate. 

Aggregation  Formula.  Workshop  participants  questioned  the  additive  aggre¬ 
gation  formula  that  was  used.  However,  no  alternative  was  suggested.  Under 
the  formula  used,  weighted  scores  are  simply  added  up  for  each  aid.  This  means 
that  if  an  aid  had  no  feasibility  at  all,  i.e.,  could  not  be  developed,  it 
could  still  get  a  high  ranking  if  it  were  rated  very  important.  One  partici¬ 
pant  suggested  that  multiplying  the  importance  score  by  the  feasibility  score 
would  give  a  ranking  closer  to  an  intuitively  correct  ranking. 

Overall  Methodology.  The  "best"  methodology  will  depend  on  the  developer's 
objectives  and  the  relationship  between  them.  MAUA  and  the  Saaty  methodology 
are  "trade  off"  methods.  That  is,  the  developer  is  willing  to  accept  a  bad 
rating  on  one  variable  in  order  to  get  a  very  good  rating  on  another  variable. 
For  example,  in  order  to  get  an  aid  with  a  great  potential  for  time  savings, 
the  developer  may  be  willing  to  accept  a  high  risk  of  development.  He  will 
accept  a  high  risk  in  order  to  get  a  higher  payoff.  This  "trade  off"  model  is 
not  appropriate  for  some  sets  of  funding  objectives.  For  example,  a  funding 
objective  may  be  to  allocate  a  certain  amount  of  money  to  fund  projects  in  each 
of  the  high,  medium,  and  low  risk  categories.  In  this  case,  including  "risk" 
as  a  MAUA  variable  would  not  correctly  model  the  developer's  funding  objec¬ 
tives.  In  this  case,  a  portfolio  model  (Sowder  &  Mandakovic,  1986)  used  in 
combination  with  a  MAUA  model  would  be  appropriate.  The  point  to  be  made  is 
that  a  MAUA  or  Saaty  prioritizing  methodology  is  not  apriori  the  best  methodol¬ 
ogy.  However,  for  this  Workshop  MAUA  appears  appropriate. 

Criteria.  The  criteria  should  be  determined  by  the  objectives  of  those  who 
will  fund  the  aids'  development.  Therefore,  subsequent  prioritization  efforts 
will  not  necessarily  use  the  same  criteria  as  those  used  in  this  Workshop.  For 
example,  if  an  objective  of  the  developers  is  to  fund  aids  which  would  contrib¬ 
ute  to  the  development  of  ES  technology,  then  this  would  be  an  additional  cri¬ 
terion. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


As  stated  previously,  the  Workshop  had  two  purposes:  (1)  to  develop  and 
prioritize  a  list  of  tactical  decision  aids  suitable  for  development  using  AI 
technology,  and  (2)  to  test  a  methodology  suitable  for  prioritizing  AI  based 
aids.  The  following  conclusions  and  recommendations  address  these  purposes. 

Prioritized  Aids.  The  four  top  ranked  aids  were  Alternate  COA  Analyzer, 
Force  Movement  Analyzer,  Obstacle  Preparation  and  Fuel  Consumption  Rates  Ana¬ 
lyzer.  All  four  had  high  interrater  agreement  and  ranked  the  same  using  both 
weighted  and  unweighted  scores.  These  results  contribute  credibility  to  the 
rankings  of  the  four  top  rated  aids.  The  four  aids  are  also  found  among  the 
top  rated  aids  of  other  priorization  efforts,  which  used  different  criteria  and 
different  candidate  lists  (e.g. ,  Archer,  Carter, &  Murray,  in  preparation;  USA 
CAORA,  1985;  Coleman  &  Miller,  in  preparation).  Although  this  priorization 
Workshop  was  a  preliminary  effort,  the  results  suggest  that  the  four  top  rated 
aids  should  be  considered  as  candidates  for  AI  based  development. 

Prioritization  Methodology.  The  following  general  recommendations  are 
made  for  a  prioritization  methodology.  These  recommendations  will  be  elabo¬ 
rated  in  a  subsequent  report  (Riedel,  in  preparation). 

1.  Prioritization  should  be  viewed  within  the  context  of  the  larger  aid 
development  process.  The  major  components  are:  (1)  requirements  analysis  to 
identify  candidate  aids,  (2)  identification  of  prioritization  criteria, 

(3)  identification  of  prioritization  model,  (4)  data  collection  including  a 
feasibility  assessment,  (5)  application  of  prioritization  model,  (6)  selection 
of  set  of  aids  for  development. 

2.  In  collecting  the  data,  the  right  data  sources  need  to  be  identified. 
The  same  raters  need  not  supply  all  the  data.  Results  will  be  no  better  than 
the  relevance  and  accuracy  of  the  data  going  into  the  prioritization  formula. 

3.  The  key  to  successful  prioritization  is  in  defining  the  appropriate 
objectives,  the  relationship  between  them,  and  identifying  criteria  to  measure 
whether  the  objectives  are  fulfilled.  The  choice  of  the  prioritization  varia¬ 
bles,  formula,  variable  weights  and  overall  methodology  should  be  made  by  those 
who  will  make  the  decisions  about  which  aids  to  fund.  These  elements  are 
determined  by  the  developer's  objectives  in  choosing  aids,  and  it  is  through 
these  elements  that  aid  selection  is  related  to  overall  management  goals. 

4.  There  are  a  variety  of  prioritization  models  available,  and  the  best 
model  depends  on  the  developer's  objectives  and  how  these  objectives  are 
related  to  one  another. 


15 


REFERENCES 


Archer,  M.A. ,  Carter,  Jr.,  C.F.,  &  Murray,  A.E.  (in  preparation).  Analysis 
of  Army  Staff  Functions.  Alexandria,  VA:  US  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences. 

Coleman,  J.P.,  &  Miller,  M.W.  (in  preparation).  Implementing  Decision  Support 
Systems  (DSS)  on  Fielded  Micro  computers.  In  S.  Riedel  &  M.  Drillings 
(Eds.).  Proceedings  of  the  Fifth  Annual  Workshop  on  Command  and  Control 
Decision  Aiding.  Alexandria,  VA:  US  Army  Research  Institute  for  the 
Behaviorial  and  Social  Sciences. 

D'Ambrosio,  B.  (1985).  Expert  systems  -  myth  or  reality?  Byte,  January, 
275-282. 

Edwards,  W.  (1977).  How  to  use  multiattribute  utility  measurement  for  social 
decision  making.  IEEE  Transactions  on  Systems.  Man,  and  Cybernetics. 

SMC-7.  326-340. 

Keeney,  R. ,  &  Raiffa,  H.  (1976).  Decisions  with  multiple  tradeoffs:  prefer¬ 
ences  and  value  tradeoffs.  New  York:  John  Wiley  &  Sons 

Liebhaber,  M.  &  Riedel,  S.  (1988).  A  survey  of  expert  system  development 

tools.  ARI  Research  Note  88-40,  Alexandria,  VA  (ADA  197-354.) 

Packard,  B.  J.,  McKeown,  P.  E.,  Fallesen,  J.  J.,  Solick,  R. ,  &  Halpln,  S.  M. 

(1987).  Design  document:  Preliminary  requirements  for  EDDIC.  ARI  Working 
Paper  LVN-87-1. 

Pitz,  G.F.,  &  McKillip,  Jr.  (1984).  Decision  analysis  for  program  evaluation. 
Beverly  Hills,  CA:  Sage  Publications. 


Riedel,  S.  L.  (in  preparation).  Prioritization  methodologies  for  AI  decision 
aids .  ARI  report. 

Saaty,  T.L.  (1980).  The  analytic  hierarchy  process.  New  York:  McGraw  Hill. 

Sowder,  VJ.  E.,  &  Mandakovic,  T.  (1986).  R&D  Project  Selection  Models. 

Research  Management,  29,  36-42. 

SPSSX  Users'  Guide.  (1983),  New  York:  McGraw  Hill  Book  Co. 

United  States  Army  Combined  Arms  Operations  Research  Activity  (CAORA).  (1985). 
G3  Analysis.  Volumes  I  and  II  (CAORA/TR-13/85).  Fort  Leavenworth,  Kansas. 


16 


APPENDIX  A 


List  of  G3  Main  Analytic  Aiding  Opportunities 

Following  is  a  list  of  53  G3  Main  analytic  aiding  opportunities  identified 
in  the  "G3  Analysis"  report  (CAORA,  1985).  The  list  shows  the  aid  names,  the 
analytic  technique(s)  to  be  employed. 

Air  Movement  Analyzer 

a.  Product  supported:  Movement  Order 

b.  Description:  This  aid  is  designed  to  analyze  whether  there  are 
sufficient  time  and  assets  to  accomplish  the  stated  mission. 

c.  Analytic  techniques:  math  model  (MM),  simulation  (SIM),  artificial 
intelligence  (AI) 

Air  Movement  Planner 

a.  Product  supported:  Movement  Order 

b.  Description:  This  aid  is  designed  to  automate  the  "stubby  pencil" 
calculations  of  an  air  movement  table. 

c.  Analytic  techniques:  MM,  optimization  technique  (OT) 

Aircraft  Asset  Analyzer 

a.  Product  supported:  Aircraft  Mission  Request  (Army  Aviation) 

b.  Description:  This  aid  is  designed  to  analyze  whether  sufficient 
aviation  assets  exist  on  mission,  time  frame,  and  priority. 

c.  Analytic  techniques:  MM,  SIM 

Aircraft  Requirements 

a.  Product  supported:  Army  Aviation  Annex 

b.  Description:  This  aid  is  designed  to  determine  number  of  aircraft 
required  to  support  the  mission. 

c.  Analytic  techniques:  MM,  AI,  SIM 

Allocate  Combat  Air  Support  (CAS)  and  Reconnaissance  (RECCE)  Aircraft 

a.  Product  supported:  Air  Request/Task  Message 

b.  Description:  This  aid  is  designed  to  determine  best  utilization  of 
CAS  and  RECCE  aircraft. 

c.  Analytic  techniques:  DA,  SIM,  AI 

Allocate  Critical  Assets  (ECM) 

a.  Product  supported:  ECM  Mission  Request 

b.  Description:  This  aid  is  designed  to  determine  the  optimal  way  to 
allocate  limited  ECM  assets. 

c.  Analytic  techniques:  SIM,  MM,  AI 
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Allocate  Replacement  Equipment,  Supplies,  and  Troops 

a.  Product  supported:  Service  Support  Annex 

b.  Description:  This  aid  is  designed  to  determine  the  best  use  of 
replacement  equipment,  supplies,  and  troops. 

c.  Analytic  Techniques:  Decision  analysis  (DA),  MM,  SIM,  OT,  AI 

Allocate  Resources 

a.  Product  supported:  Engineer  Mission  Coordination  Sheet 

b.  Description:  This  aid  is  designed  to  allocate  critical  resources 
within  the  Engineer  functional  area. 

c.  Analytic  techniques:  MM,  SIM,  DA,  Al 

Assign  Critical  Replacement  Units,  Personnel,  and  Materiel 

a.  Product  supported:  Allocate/Prioritize  Replacement  Personnel, 
Meteriel,  and  Units 

b.  Description:  This  aid  is  designed  to  make  allocations  based  on  need 
and  is  similar  to  aid  3-6,  above. 

c.  Analytic  techniques:  DA,  MM,  SIM,  OT,  AI 

Assign  PSYOP  Assets 

a.  Product  supported:  Psychological  Operations  Annex 

b.  Description:  This  aid  is  designed  to  optimally  assign  psychological 
elements  and  equipment. 

c.  Analytic  techniques!  DA,  AI,  MM,  OT,  SIM 

Basic  Load  Allocations 

a.  Product  supported:  OPORD  (Service  Support) 

b.  Description:  This  aid  is  designed  to  determine  basic  load  based  on 
ammunition  availability,  intensity  of  conflict,  and  resupply. 

c.  Analytic  techniques:  MM,  AI,  SIM 

Chemical  Effects  Prediction 

a.  Product  supported:  Chemical  Strike  Warning 

b.  Description:  This  aid  is  designed  to  determine  potential  effects  and 
recommended  actions  to  minimize  those  effects. 

c.  Analytic  techniques:  MM 

Combat  Effectiveness  (Obstacles) 

a.  Product  supported:  Engineer  Spot  Report 

b.  Description:  This  aid  is  designed  to  evaluate  the  effectiveness  of 
barriers  and  obstacles. 

c.  Analytic  techniques:  MM,  OT 
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Compare  Alternative  Courses  of  Action 

a.  Product  supported:  Operations  Estimate 

b.  Description:  This  aid  is  designed  to  analyze  alternatives. 

c.  Analytic  Techniques:  DA,  AI,  MM,  OT,  SIM 

Control  Procedures/Status 

a.  Product  supported:  Airspace  Management  Annex 

b.  Description:  This  aid  is  designed  to  analyze  various  control  methods. 

c.  Analytic  techniques:  DA,  SIM,  AI 

Controlled  Supply  Rate  (CSR) 

a.  Product  supported:  OPORD  (Service  Support) 

b.  Description:  This  aid  is  designed  to  analyze  ammunition  expenditure 
rates,  and  where  required,  recommend  control  rate  restrictions. 

c.  Analytic  techniques:  MM 

Damage  Analysis  (Effects  on  Enemy) 

a.  Product  supported:  Post-Strike  Analysis  (Nuclear) 

b.  Description:  This  aid  is  designed  to  evaluate  the  impact  of  the 
damage  and  the  various  options  of  repair. 

c.  Analytic  techniques:  DA,  SIM 

Denial  Preparation 

a.  Product  supported:  Engineer  Annex-Denial  Appendix 

b.  Description:  This  aid  is  designed  to  prioritize  the  placement  of 
appropriate  obstacles. 

c.  Analytic  techniques:  OT,  MM,  SIM,  AI,  DA 
Determine  Replacement  Priorities 

a.  Product  supported:  OPORD  (Service  Support) 

b.  Description:  This  aid  is  designed  to  assign  replacement  priorities 
based  on  mission,  strength,  and  location. 

c.  Analytic  techniques:  DA,  SIM,  AI 

Evaluate  Damage  Repair  Alternatives 

a.  Product  supported:  Engineer  Report  (Damage) 

b.  Description:  This  aid  is  designed  to  evaluate  the  impact  of  the 
damage  and  the  various  options  of  repair. 

c.  Analytic  techniques;  DA,  SIM 

Expenditure  Rates 

a.  Product  supported:  Fire  Support  Annex 

b.  Description:  This  aid  is  designed  to  determine  expenditure  rates  and, 
when  necessary,  CSR  based  on  mission  and  unit. 

c.  Analytic  techniques:  MM,  OT,  DA,  SIM 
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Fallout  Prediction 


a.  Product  supported:  Post-Strike  Analysis  (Nuclear) 

b.  Description:  This  aid  is  designed  to  predict  fallout  as  a  result  of  a 
nuclear  strike. 

c.  Analytic  techniques:  MM 

Force  Movement  Analyzer 

a.  Product  supported:  OPORD  (Execution)_ 

b.  Description:  This  aid  is  designed  to  investigate  force  movement 
alternatives  and  time  required  for  force  movement. 

c.  Analytic  techniques:  MM,  SIM 

Forecast  Tube  Replacement 

a.  Product  supported:  Artillery  Situation  Report 

b.  Description:  This  aid  is  designed  to  forecast  artillery  tube 
replacement  requirements  based  on  current  status  and  future  mission. 

c.  Analytic  techniques:  MM 

Forecast  Unit  Status 

a.  Product  supported:  Project  Unit  Status 

b.  Description:  This  aid  is  designed  to  project  unit  status  based  on 
mission,  current  status,  and  environment  factors;  when  necessary,  it  would 
activate  a  critical  situation  alert. 

c.  Analytic  technique:  MM 

Forecast  Usage  Rates 

a.  Product  supported:  Required  Ammunition  Supply  Rate  Report 

b.  Description:  This  aid  is  designed  to  forecast  ammunition  usage  based 
on  mission  and  unit  status. 

c.  Analytic  techniques:  MM,  SIM 

Fuel  Consumption  Rates 

a.  Product  supported:  OPORD  (Service  Support) 

b.  Description:  This  aid  is  designed  to  determine  fuel  requirements 

based  on  type  vehicles,  mission,  terrain,  weather,  etc. 

c.  Analytic  techniques:  MM 

Hazard  Areas 

a.  Product  supported:  NBC  5  (Report  of  Areas  of  Actual  Contamination) 

b.  Description:  This  aid  is  designed  to  determine  actual  contamination 
areas . 

c.  Analytic  techniques;  MM,  AI 
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Integrated  CAS  with  Fire  Support  Plan 

a.  Product  supported:  Air  Request/Task  Message  (Pre-planned) 

b.  Description:  This  aid  is  designed  to  integrate  close  air  support  with 
overall  fire  support  plan. 

c.  Analytic  techniques:  DA,  Al,  SIM 

NBC  Effects  Evaluation 

a.  Product  supported:  NBC  2  (Evaluated  Data  Report) 

b.  Description:  This  aid  is  designed  to  evaluate  NBC  strike  data. 

c.  Analytic  techniques:  MM,  SIM 

Nuclear  Effects  Preparation 

a.  Product  supported:  Nuclear  Strike  VJarning 

b.  Description;  This  aid  is  designed  to  determine  potential  effects  and 
recommended  action  to  minimize  those  effects. 

c.  Analytic  techniques:  Mil 

Obstacle  Emplacement  Plan 

a.  Product  supported:  Engineer  Barrier  Report 

b.  Description:  This  aid  is  designed  to  optimally  select  types  and 
locations  of  obstacles. 

c.  Analytic  techniques:  OT,  MM,  SlM,  DA,  AI 

Obstacle  Preparation 

a.  Product  supported:  Engineer  Annex  -  Obstacle  Appendix 

b.  Description:  This  aid  is  designed  to  prioritize  work  based  on 
critical  resources. 

c.  Analytic  techniques:  OT,  MM,  SIM,  DA,  AI 
Operational  Effectiveness 

a.  Product  supported:  Psychological  Operations  Annex 

b.  Description:  This  aid  is  designed  to  estimate  the  operational 
effectiveness  of  a  given  PSYOP  course  of  action. 

c.  Analytic  techniques:  DA,  MM 

Optimal  Atomic  Demolition  Munitions  (ADM)  Employment 

a.  Product  supported:  Engineer  Annex  -  ADM  Appendix 

b.  Description:  This  aid  is  designed  to  optimally  select  and  emplace 

ADM. 

c.  Analytic  techniques:  OT,  MM,  SIM,  DA,  Al 
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Optimal  Friendly  Employment  (EW) 

a.  Product  supported;  Electronic  Warfare  Annex 

b.  Description:  This  aid  is  designed  to  optimally  employ  electronic 
warfare  assets. 

c.  Analytic  techniques:  SIM,  AI,  OT,  DA 
Organize  for  Combat  (FS) 

a.  Product  supported:  Fire  Support  Annex 

b.  Description:  This  aid  is  designed  to  effectively  organize  for  combat. 

c.  Analytic  techniques:  DA,  AI,  MM 

Predict  Contamination  (ID  Affected  Units) 

a.  Product  supported:  NBC  3  (Immediate  Warning  of  Expected 
Contamination) 

b.  Description:  This  aid  is  designed  to  calculate  expected  hazard  area 
and  determine  affected  units. 

c.  Analytic  techniques:  MM,  SIM 

Pre-position  Decon  Supplies 

a.  Product  supported:  NBC  Defense  Annex 

b.  Description:  This  aid  is  designed  to  ascertain  the  best  location  and 
quantities  for  pre-positioning. 

c.  Analytic  techniques:  DA,  SIM,  AI 

Prescribed  Chemical  Load  (PCL) 

a.  Product  supported:  Fire  Support  Annex 

b.  Description;  This  aid  is  designed  to  allocate  chemical  munitions 
based  on  availability,  mission,  and  release  policy. 

c.  Analytic  techniques:  MM,  DA,  AI 

Prescribed  Nuclear  Load  (PNL) 

a.  Product  supported:  Fire  Support  Annex 

b.  Description:  This  aid  is  designed  to  allocate  nuclear  munitions  based 
on  availability,  mission,  and  release  policy. 

c.  Analytic  techniques:  MM,  DA,  AI 

Priorities  of  Fire 

a.  Product  supported:  Fire  Support  Annex 

b.  Description:  This  aid  is  designed  to  assign  priorities  of  fire  to 
field  artillery  units. 

c.  Analytic  techniques:  DA,  SIM,  OT,  AI 
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Pi iorities/Allocation  (ADA) 


a.  Product  supported:  Air  Defense  Annex 

b.  Description:  This  aid  is  designed  to  establish  weapon  control 
procedures  and  allocate  weapon  systems. 

c.  Analytic  techniques:  DA,  MM,  SIM,  DA,  AI 

Rear  Area  Protection  Capabilities 

a.  Product  supported:  Rear  Area  Protection  Annex 

b.  Description:  This  aid  is  designed  to  evaluate  rear  area  protection 
plans  and  identify  assets  for  the  rear  battle. 

c.  Analytic  techniques:  SIM,  MM,  DA 

Relative  Combat  Power 

a.  Product  supported:  Operations  Estimate 

b.  Description:  This  aid  is  designed  to  estimate  friendly  and  relative 
combat  power. 

c.  Analytic  techniques:  MM 
Route  Evaluation  (AVN) 

a.  Product  supported:  Aircraft  Mission  Request  (Army  Aviation) 

b.  Description:  This  aid  is  designed  to  evaluate  a  selected  flight 
route  in  terms  of  risk  and  protection. 

c.  Analytic  techniques:  DA,  AI,  MM 

Target  Allocation 

a.  Product  supported:  Chemical  Support  Annex 

b.  Description:  This  aid  is  designed  to  select  chemical  targets  based  on 
priority  and  chemical  munitions  availability. 

c.  Analytic  techniques:  SIM,  AI,  OT,  DA 

Target  Susceptibility 

a.  Product  supported:  NBC  Defense  Annex 

b.  Description:  This  aid  is  designed  to  evaluate  friendly  units' 
susceptibility  to  an  enemy  NBC  strike. 

c.  Analytic  techniques:  DA,  SIM,  AI 

Task  Organization 

a.  Product  supported:  OPORD  (Task  Organization) 

b.  Description:  This  aid  is  designed  to  organize  combat  and  combat 
support  units  for  combat  based  on  mission,  terrain,  unit  status,  etc. 

c.  Analytic  techniques:  DA,  AI,  MM 

Terrain  Management 

a.  Product  supported:  OPORD  (Execution) 

b.  Description:  This  aid  is  designed  to  assign  units  to  terrain. 

c.  Analytic  techniques:  MM,  OT,  SIM 
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Time  Analyzer 


a.  Product  supported:  Warning  ORder 

b.  Description:  This  aid  is  designed  to  time-sequence  critical  actions 
to  ensure  subordinate  units  have  time  to  execute. 

c.  Analytic  techniques:  MM,  DA 

Troop  Exposure 

a.  Product  supported:  NBC  Defense  Annex 

b.  Description:  This  aid  is  designed  to  evaluate  and  monitor  NBC  status 
of  units. 

c.  Analytic  techniques:  MM 

Unit  Movement  Planner 

a.  Product  supported:  Movement  Order 

b.  Description:  This  aid  is  designed  to  plan  and  publish  movement  orders 
for  units,  brigade,  and  below. 

c.  Analytic  techniques:  MM,  SIM,  AI,  OT 
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Appendix  B 


Analytic  Aiding  Opportunities  (adjusted  rank  order)l 


Adjusted 


Absolute 

Adjusted 

Feasi¬ 

AID 

Aid 

Adjusted 

Raw  Rank 

Importance 

bility 

DESCRIPTOR 

ID# 

Rank 

Rank  Diff 

Rank 

Rank 

Unit  Movement  Planner 

3-51 

1 

1 

0 

1 

12 

Force  Movement  Analyzer 

3-24 

2 

3 

1 

2 

11 

Air  Movement  Analyzer 

3-04 

3 

4 

1 

4 

19 

Fuel  Consumption  Rates 

3-26 

4 

5 

1 

7 

10 

Air  Movement  Planner 

3-05 

5 

2 

3 

13 

3 

Assign  Critical  Replace- 

3-08 

6 

6 

0 

11 

13 

ment  Units,  Personnel, 
and  Materiel 

Terrain  Management 

3-46 

7 

12 

5 

10 

29 

Denial  Preparation 

3-19 

8 

22 

14 

12 

30 

Time  Analyzer 

3-47 

9 

7 

2 

28 

4 

Pre-Position  Decontami- 

3-18 

10 

17 

7 

15 

27 

nation  of  Supplies 

Compare  Alternate  Courses 

3-13 

11 

42 

31 

3 

49 

of  action 

Obstacle  Preparation 

3-31 

12 

14 

2 

17 

28 

Predict  Contamination 

3-39 

13 

9 

4 

36 

2 

(ID  Affected  Units) 

Forecast  Unit  Status 

3-52 

14 

30 

16 

6 

40 

Chemical  Effects 

3-30 

15 

18 

3 

23 

18 

Prediction 

Expenditure  Rates  (FS) 

3-22 

16 

11 

5 

33 

6 

Basic  Load  Allocations 

3-10 

17 

8 

9 

42 

1 

Nuclear  Effects  Predic- 

3-21 

18 

16 

2 

30 

15 

tion 

Aircraft  Asset  Analyzer 

3-02 

19 

10 

9 

26 

21 

Priorities  of  Fire  (FS) 

3-40 

20 

24 

4 

18 

32 

Priori ties /Allocation 

3-38 

21 

33 

12 

9 

43 

(ADA) 

Rear  Area  Protection 

3-41 

22 

39 

17 

8 

45 

Capabilities 


1  "G3  Analysis"  Report 
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Analytic  Aiding  Opportunities  (adjusted  rank  order) 
(CONTINUED 


Adjusted 

Absolute  Adjusted  Feasi- 


AID 

Aid 

Adjusted 

Raw 

Rank 

Importance 

bility 

DESCRIPTOR 

ID// 

Rank 

Rank 

Diff 

Rank 

Rank 

Troop  Exposure  (NBC) 

3-50 

23 

19 

4 

35 

17 

Evaluate  Damage  Repair 

3-17 

24 

27 

3 

21 

31 

Alternatives 

Forecast  Tube  Replace- 

3-25 

25 

13 

12 

39 

9 

ment  (FS) 

Forecast  Usage  Rates 

3-53 

26 

23 

3 

32 

23 

(RSR) 

Allocate  CAS  and  RECCE 

3-11 

27 

28 

1 

31 

25 

Controlled  Supply  Rate 

3-15 

28 

13 

15 

43 

7 

(CSR) 

Route  Evaluation  (AVN) 

3-44 

29 

35 

6 

5 

52 

ADM  Employment 

3-33 

30 

29 

1 

38 

20 

Task  Organization 

3-45 

31 

31 

0 

22 

34 

Target  Allocation 

3-48 

32 

25 

7 

40 

16 

(Chemical) 

Aircraft  Requirements 

3-03 

33 

32 

1 

25 

35 

Prescribed  Nuclear 

3-37 

34* 

26 

8 

37 

22 

Load  (RNL) 

Prescribed  Chemical 

3-36 

34* 

26 

8 

37 

22 

Load  (PCL) 

Optimal  Friendly  Employ- 

3-34 

35 

36 

1 

14 

44 

ment  (EW) 

Organize  for  Combat  (FS) 

3-35 

36 

37 

1 

24 

36 

Allocate  Engineer 

3-07 

37 

15 

22 

41 

26 

Resources 

*Ties  were  allowed  for  ranks.  PCL  and  PNL  had  a  tie  for  all  scoring  schemes. 
Therefore,  the  adjusted  ranks  ranged  from  1-52  for  a  total  of  53  aiding 
opportunities . 
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Analytic  Aiding  Opportunities  (adjusted  rank,  order) 
(CONTINUED) 


Adjusted 

Absolute  Adjusted  Feasi- 


AID 

Aid 

Adjusted 

Raw 

Rank 

Importance 

bility 

DESCRIPTOR 

ID# 

Rank 

Rank 

Diff 

Rank 

Rank 

Target  Susceptibility 

3-49 

38 

34 

4 

27 

37 

(NBC) 

Fallout  Prediction 

3-23 

39 

21 

18 

48 

8 

(Nuclear) 

Hazard  Areas  (NBC) 

3-27 

40 

20 

20 

45 

14 

Allocate  Replacements 

3-06 

41 

43 

2 

16 

48 

Obstacle  Emplacement  Plan 

3-30 

42 

41 

1 

29 

39 

Integrate  CAS  (FS) 

3-28 

43 

44 

1 

20 

42 

Relative  Combat  Power 

3-42 

44 

45 

1 

34 

38 

Control  Procedure  (A2C2) 

3-14 

45 

46 

1 

19 

51 

NBC  Effects  Evaluation 

3-29 

46 

38 

8 

47 

24 

Post-Strike  Analysis 

3-16 

47 

31 

16 

50 

5 

(Nuclear) 

Determined  Replacement 

3-43 

48 

47 

1 

44 

41 

Priorities 

Allocate  Critical  Assets 

3-01 

49 

40 

9 

51 

33 

(ECM) 

Assign  PSYOP  Assets 

3-09 

50 

49 

1 

46 

47 

Obstacle  Effectiveness 

3-12 

51 

48 

3 

49 

46 

PSYOP  Effectiveness 

3-32 

52* 

50 

2 

52 

50 

*  Ties  were  allowed  for  ranks.  PCL  and  PNL  had  a  tie  for  all  scoring  schemes. 
Therefore,  the  adjusted  ranks  ranged  from  1-52  for  a  total  of  53  aiding 
opportunities. 
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APPENDIX  C 

SAMPLE  AID  PRIORITIZATION  WORKSHgET 

1.  AID  Ksaunoi:  Jcu4J^ 

2.  AID.NUHUK:  5>V5 

3.  fWOUCT  swpomo:  OPOif^  CJa^ 

4.  PRINART  AKALYTIC  TtOmiQUC: 

s.  swraftTiNS  AXALnic  T£CMNiQuc(s}: 

ft.  TASR(S)  sumwia  (bt  mraau  /o.^  /  2^ 

7.  cent  swramo  crital  MMCX):  // 

8.  BRIEF  AID  OESaumWI:  ^  *♦ 

Ct^rfUct  u^0ttAt  C^nkat"  ^  rrH*  t  <  rx 

I.  iuu!ffm*i?uu2s'7w^M^  Bw  Au 

4.  FraRMWy  (low.  MsAl  fflK.  J2L.  SBBL 

• . V.s  ./09 

0 S  20 

b.  TIm  and  Quality  Savioga  ISaall, 'Urgi} 

0  ;  •  s  Tv  10 

e.  CC!ll  (Fw,  Naqy) 

. jT  . . . .  >g.g  .3AQ  ],I9S 

O'  '  S  20 

20.  lAloaiS  BT  SCALES  (FEASIBIUH): 

4.  Oporatlonal  (Lou.  Nigh) 

^  *  '-'X'  r  L  .  190 

0  8  '  20 

b.  CcDneBieil  (High  Cost.  Low  Coot) 

V.  *7  .0^7  ■ 

0  20 

e.  Todwlcal  (High  Risk,  Low  Mak) 

L.'  •  ».L  -f.!. '  I . 1  ”,.’.U, "  ■  ■  . I . ,>.LL|  jLJL  .aSif  Jl^ 

0  s  ./\  20 

TOTAL  SCaiO:  3r.?  /.OQU  V. 


EKalAI: 
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APPENDIX  D 


1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9 

10. 

11. 

12. 

13. 

14. 

15. 

16. 

17. 

18. 

19. 

20. 
21. 
22. 
23. 


INTERRATER  AGREEMENT  ON  RATINGS  OF  AID  IMPORTANCE  AND  FEASIBILITY 


Aid 


Chronbach's  alpha 


Prescribed  Nuclear  Load  0 

Rear  Area  Protection  Capabilities  .724 

Situation  Assessment  .791 

NBC  Effects  Evaluation  .449 

Air  Movement  Planner  0 

Prescribed  Chemical  Load  (PCL)  0 

Air  Movement  Analyzer  0 

Nuclear  Effects  Prediction  .3191 

Priorities/Allocation  (ADA)  .374 

Compare  Alternative  Courses  of  Action  .875 

Force  Movement  Analyzer  .617 

Expenditure  Rates  (FS)  0 

Assign  PSYOP  Assets  0 

Obstacle  Preparation  .610 

Chemical  Effects  Prediction  .096 

Fuel  Consumption  Rates  .725 

Basic  Load  Allocations  0 

Optimal  Atomic  Demolition  Munitions 

Employment  .230 

Aircraft  Asset  Analyzer  0 

Controlled  Supply  Rate  (CSR)  .080 

Fallout  Prediction  (Nuclear)  0 

Forecast  Tube  Replacement  .706 

Denial  Preparation  .041 


Number  of  Raters 
7 
9 
9 
9 
6 
7 
7 
9 
7 

10 

7 

9 

7 
9 
9 

8 
8 

9 

9 

8 

8 

7 

10 


29 


